Method and device for generating and detecting fingerprints for synchronizing audio and video

ABSTRACT

This invention relates to a device and a method of generating a first and a second fingerprint ( 102,104 ) usable for synchronization of at least two signals ( 101,103 ) and corresponding method and device for synchronizing two or more signals. A fingerprint pair is generated on the basis of a segment of a first signal e.g. an audio signal and of a segment of a second signal e.g. a video signal at each synchronization time point. The generated fingerprint pair(s) are stored in a database ( 203 ) and communicated or distributed to a synchronization device ( 303 ). During synchronization, fingerprint(s) of the audio signal and fingerprint(s) of the video signal to be synchronized are generated and matched against fingerprints in the database. When a match is found, the fingerprints also determine the synchronization time point, which is used to synchronize the two signals. In this way, a simple, reliable and efficient way of synchronizing at least two signals is obtained. Further, this is enabled without modifying either the first and second signal (or subsequent signals). The signals may even be distorted or changed to some extent while still enabling synchronization.

RELATED APPLICATIONS

This application is a Continuation of U.S. application Ser. No.10/566,003, filed on Jan. 20, 2006, which is a U.S. national stagefiling of PCT/IB2004/051259, filed on Jul. 20, 2004 and published onFeb. 3, 2005 as WO 2005/011281, which claims priority to EP ApplicationNo. 03102301.3, filed on Jul. 25, 2003, which applications andpublication are incorporated herein by reference in their entirety.

BACKGROUND OF THE INVENTION

The present invention relates to synchronisation between at least twosignals. More specifically, the invention relates to a method, and acorresponding device, of synchronising a first signal, e.g. an audiosignal, and a second signal, e.g. a video signal. The invention alsorelates to a method, and a corresponding device, of enablingsynchronisation of an audio signal and a video signal. Further, theinvention relates to a computer readable medium having stored thereoninstructions for causing one or more processing units to execute themethod according to the invention.

Synchronisation of a video stream with a corresponding audio stream is adifficult problem which has received a lot of attention. Many solutionsto this problem have been proposed and implemented. Most of thesesolutions require manual synchronisation by a skilled operator.Typically the operator looks for visual clues within the picture todetermine if the sound heard corresponds to the picture and that theyare indeed synchronous. The problem becomes much harder when thesynchronisation needs to be done automatically. This is a problem thatis becoming more and more relevant, as processing and distribution ofaudio and video signals are becoming ever more complicated, both insideand outside a studio environment. An example of the latter is thefollowing: A consumer records a movie with his video recorder. He wouldlike to view it with the original sound-track. Therefore he buys theoriginal soundtrack, which, for example, is streamed to him over theInternet. Now the audio and the video need to be synchronisedautomatically e.g. in/by his video recorder or another synchronisationdevice.

One previous system that would allow the automatic synchronisation of anaudio and a video stream is marketed by Tektronix. In this system, theenvelope of the audio signal is embedded by means of a watermark intothe video signal. At any point in the distribution of processing chain,the actual audio envelope can be compared to the embedded one, fromwhich the delay between the two streams can be derived. Subsequently,the delay of the audio is corrected to achieve synchronisation. However,this system requires the co-operation of the broadcaster or anotherdistributor, because before transmission, the watermark needs to beembedded in the video. Further, this system can only associate oneparticular audio stream to the video. Once the envelope of an audiostream has been embedded, the system can only synchronise the video withthat particular audio stream. For synchronising other audio streams,another watermark should have been embedded. Finally, the system isrestricted to synchronisation between an audio stream and a videostream.

SUMMARY OF THE INVENTION

It is an object of the invention to provide a method and correspondingdevice for generating a first and a second fingerprint usable forsynchronisation of at least two signals and corresponding method anddevice for synchronising two or more signals that solves theabove-mentioned problems. A further object is to provide this in asimple and efficient way. Another object is to enable simple, reliableand accurate localisation of a given part of a multimedia signal. Afurther object is to enable automatic synchronisation between a firstsignal and at least a second signal without modifying any of thesignals.

This is achieved by a method (and corresponding device) of enablingsynchronisation of a first and a second signal, the method comprisingthe steps of

-   -   deriving a first fingerprint on the basis of a segment of the        first signal, where the segment of the first signal is        unambiguously related with a first synchronisation time point,    -   deriving a second fingerprint on the basis of a segment of the        second signal, where the segment of the second signal is        unambiguously related with a second synchronisation time point,        and    -   supplying the first and second fingerprints to a synchronisation        device    -   and by a method (and corresponding device) of synchronising two        or more signals, the method comprising the steps of:    -   generating a first fingerprint stream on the basis of a first        signal,    -   generating a second fingerprint stream on the basis of a second        signal,    -   comparing a segment of the first fingerprint stream with one or        more first fingerprints stored in at least one database in order        to determine if a match exists or not,    -   comparing a segment of the second fingerprint stream with one or        more second fingerprints stored in the at least one database in        order to determine if a match exists or not, and    -   if a match exists for both a first and a second fingerprint        determining a location of a first synchronisation time point for        the first signal and a location of a second synchronisation time        point for the second signal and synchronising the first and the        second signal using the determined locations.

In this way, a simple, reliable and efficient way of synchronising atleast two signals is obtained. Further, this is enabled withoutmodifying either the first and second signal (or subsequent signals).The signals may even be distorted or changed to some extent while stillenabling accurate synchronisation due to the use of fingerprints.

A fingerprint of a multimedia object/content/signal is a representationof perceptual features of the object/content/signal part in question.Such fingerprints are sometimes also known as “(robust) hashes”. Morespecifically, a fingerprint of a piece of audio or video is anidentifier which is computed over that piece of audio or video and whichdoes not substantially change even if the content involved issubsequently transcoded, filtered or otherwise modified.

Advantageous embodiments of the methods and devices according to thepresent invention are defined in the sub-claims.

Further, the invention also relates to a computer readable medium havingstored thereon instructions for causing one or more processing units toexecute the method according to the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 a schematically illustrates generation of fingerprint pair(s) tobe used for synchronisation between an audio and a video signal;

FIG. 1 b schematically illustrates detection of such generatedfingerprint pair(s) used for synchronisation according to the presentinvention.

FIG. 2 illustrates a schematic block diagram of a fingerprint generationdevice according to the present invention;

FIG. 3 illustrates a schematic block diagram of a synchronisation devicedetecting and using fingerprints according to the present invention;

FIG. 4 illustrates one example of tables/records according to thepresent invention;

FIG. 5 illustrates an alternative embodiment of a relationship betweentime points in a first and in a second signal.

FIG. 6 illustrates an embodiment where first and second representationsare stored at a remote location;

FIG. 7 illustrates schematically more specifically how thesynchronisation in one embodiment may be done in a synchronisationdevice using buffers.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 a schematically illustrates generation of fingerprint pair(s) tobe used for synchronisation between an audio and a video signal.

Shown are a digital or analog first signal 101 and a digital or analogsecond signal 103. In the following the first signal 101 is an audiosignal and the second signal 103 is a video signal.

At one or more synchronisation time points T_(n), T_(n+1) a fingerprintpair has to be derived. These time points are selected according to atleast one predetermined criteria. E.g. criteria specifying a time pointat the beginning of the audio and/or video signal, a time point at theend and a time point in-between. Alternatively, the time points may beselected according to: one at the beginning and one time point for eachpoint after a given period of time have lapsed, e.g. one time point forevery 2 minutes or every 2 seconds, etc. Alternatively, the time pointsmay be derived from analysis of the underlying signal itself, e.g. ateach scene change in a video signal. Just a single synchronisation timepoint T_(n), T_(n+1) is needed in order to enable a synchronisationbetween the two signals 101, 103 according to the present invention.However, the use of more time points T_(n), T_(n+1) enables a bettersynchronisation e.g. in a situation where one (or both) of the signalshave been truncated, modified, etc. One example taking advantage ofseveral time points could e.g. be when a user has recorded a movie andhas bought the original soundtrack as described earlier but where themovie has been recorded with commercial breaks. By adding moresynchronisation time points a better synchronisation is enabled,especially if the synchronisation time points is at or near the end timepoints of the commercial breaks.

One audio fingerprint 102 is derived for each synchronisation time pointT_(n), T_(n+1) for the audio signal 101 and a video fingerprint 104 isderived for the video signal 103 at the same synchronisation timepoint(s) T_(n), T_(n+1) resulting in a fingerprint pair 102, 104 foreach synchronisation time point T_(n), T_(n+1). A fingerprint (for bothaudio and/or video) for a given time point T_(n), T_(n+1) is preferablyderived on a segment of the signal where the segment (substantially)starts at the given time point. Alternatively, the segment may end(substantially) at the given time point T_(n); T_(n+1), or the segmentmay start or end at a predetermined distance (substantially) before orafter the given time point T_(n); T_(n+1), or the given time pointT_(n); T_(n+1) may be at a predetermined time point between a start andan end of the segment or any other scheme as long as the same scheme isapplied during synchronisation to determine the given time point T_(n);T_(n+1) on the basis of a fingerprint as will be explained in moredetail in connection with FIG. 1 b.

The size of the fingerprints may both be of a predetermined fixed sizeor alternatively of a variable size.

One method for computing a robust fingerprint is described ininternational patent application WO 02/065782, although of course anymethod for computing a robust fingerprint can be used.

European patent application 01200505.4 describes a method that generatesrobust fingerprints for multimedia content such as, for example, audioclips, where the audio clip is divided in successive (preferablyoverlapping) time intervals. For each time interval, the frequencyspectrum is divided in bands. A robust property of each band (e.g.energy) is computed and represented by a respective fingerprint bit.

Multimedia content is thus represented by a fingerprint comprising aconcatenation of binary values, one for each time interval. Thefingerprint does not need to be computed over the whole multimediacontent, but can be computed when a portion of a certain length has beenreceived. There can thus be plural fingerprints for one multimediacontent, depending on which portion is used to compute the fingerprintover.

Further, video fingerprinting algorithms are known, e.g. from thefollowing disclosure: Job Oostveen, Ton Kalker, Jaap Haitsma: “FeatureExtraction and a Database Strategy for Video Fingerprinting”. 117-128.IN: Shi-Kuo Chang, Zhe Chen, Suh-Yin Lee (Eds.): Recent Advances inVisual Information Systems, 5th International Conference, VISUAL 2002Hsin Chu, Taiwan, Mar. 11-13, 2002, Proceedings. Lecture Notes inComputer Science 2314 Springer 2002.

According to the present invention, an audio fingerprint 102 and a videofingerprint 104 are generated for each time point T_(n), T_(n+1) on thebasis of a given segment of the audio signal 101 and a segment of thevideo signal 103 at or near the specific time point.

In this way, a given fingerprint pair 102, 104 is a synchronisationmarker enabling a very accurate and very precise location of a giventime point of the signals 101 and 103 without using the specific timepoint but instead using (a segment of) the signal. Further, this isenabled without changing the signals. Even for video fingerprinting thelocalisation is typically frame accurate, at least as long as anydistortion of the video signal is not too severe.

After a fingerprint pair 102, 104 has been generated it is preferablystored for later use in a database, memory, storage and/or the like.

There are several advantages in storing fingerprint pairs (102, 104 formultimedia signals 101, 103 in a database instead of the multimediasignals itself. To name a few:

The memory/storage requirements for the database are reduced.

The comparison of fingerprints is more efficient than the comparison ofthe multimedia signals themselves, as fingerprints are substantiallyshorter than the signals.

Searching in a database for a matching fingerprint is more efficientthan searching for a complete multimedia signals, since it involvesmatching shorter items.

Searching for a matching fingerprint is more likely to be successful, assmall changes to a multimedia signal (such as encoding in a differentformat or changing the bit rate) do not affect the fingerprint.

The generated fingerprint pairs 102, 104 stored in the database may thenbe distributed to one or more synchronisation devices (via the Internetor via other means) for synchronisation of the signals according to thepresent invention e.g. before playback, storage, further transmission ofboth (synchronised) signals, etc.

Note that the invention is also applicable to synchronisation of morethan two signals and also to signals being other types of signal thanaudio and video, as long as a robust fingerprint may be obtained. Inprincipal any number of signals may be synchronised according to thepresent invention. This would simply require an additional fingerprintat each time point T_(n), T_(n+1) for each additional signal.

Alternatively, the fingerprint pair may also be generated at differenttime points for the respective signals, i.e. one fingerprint of thefingerprint pair may be generated e.g. at 25 seconds of the first signalwhile the other fingerprint may be generated e.g. at 30 seconds of thesecond signal. However, this requires a well-defined relationshipbetween with each respective time point (e.g. 25 seconds and 30 secondsin the above example) to a common time line/frame. This alternativeembodiment will be described in greater detail in connection with FIG.5.

FIG. 1 b schematically illustrates detection of such generatedfingerprint pair(s) used for synchronisation according to the presentinvention. Shown are a digital or analog first (to-be-synchronised)signal 101 and a digital or analog second (to-be-synchronised) signal103. In the following the first signal (101 is an audio signal and thesecond signal 103 is a video signal. Further shown are a firstfingerprint stream 105 and a second fingerprint stream 106 that aregenerated continuously or substantially continuously on the basis of theaudio signal 101 and the video signal 103, respectively. Alternatively,the fingerprint streams 105, 106 are generated in segments. Eachfingerprint stream 105, 106 (or segments) is compared with fingerprints102, 104, e.g. stored in a database, in order to determine if there is amatch or not. More specifically, the audio fingerprint stream 105 iscompared with stored audio fingerprints 102 and the video fingerprintstream 106 is compared with stored video fingerprints 104. The storedfingerprints 102, 104 are generated as explained in connection with FIG.1 a e.g. at a central location. The stored fingerprints 102, 104 aree.g. received via the Internet or via some other means e.g. from thecentral location.

When a match between a segment of the audio fingerprint stream 105 and agiven audio fingerprint 102 in the database is found and a match betweena segment of the video fingerprint stream 106 and a given videofingerprint 104 in the database is found, i.e. when a matchingfingerprint pair have been found, the appropriate synchronisation timepoint T_(n); T_(n+1) is also given when the fingerprints 102, 104 havebeen generated according to the present invention and as explained inconnection with FIG. 1 a.

The specific synchronisation time point T_(n); T_(n+1) is determineddependent on the scheme that has been used during generation of theaudio fingerprint 102 and the video fingerprint 104 at that particulartime point T_(n); T_(n+1).

Preferably, the specific synchronisation time point T_(n); T_(n+1) isgiven by letting the segment of the audio signal 101 and the segment ofthe video signal 103 that the matching fingerprint pair 102, 104originally has been based on during generation (according to FIG. 1 a)starting (substantially) at the given time point T_(n); T_(n+1). Inalternative embodiments, the segment of the audio signal 101 and thesegment of the video signal 103 end (substantially) at the given timepoint T_(n); T_(n+1), the segments of the audio and video signals 101,103 are starting or ending at a predetermined distance before or afterthe given synchronisation time point T_(n); T_(n+1) or the givensynchronisation time point T_(n); T_(n+1) may be at a predetermined timepoint between a start and an end of the segments of the audio signal 101and the video signal 103.

The synchronisation device simply needs to be aware of the relationshipbetween a given fingerprint and the given time point used duringgeneration, which may be determined and implemented during manufactureof the synchronisation device or alternatively be updatable.

As explained, after a matching fingerprint pair 102, 104 is determined,the time point T_(n); T_(n+1) of this pair is also know and serves as asynchronisation time point as this time point directly gives a referencepoint between the two signals 101 and 103. The synchronisation devicethen compensates for the delay (if any) between the two signals e.g. byshifting one of them so that they are aligned with respect to the timepoint.

The above-mentioned international patent application WO 02/065782describes various matching strategies for matching fingerprints computedfor an audio clip with fingerprints stored in a database. One suchmethod of matching a fingerprint representing an unknown informationsignal with a plurality of fingerprints of identified informationsignals stored in a database to identify the unknown signal usesreliability information of the extracted fingerprint bits. Thefingerprint bits are determined by computing features of an informationsignal and thresholding said features to obtain the fingerprint bits. Ifa feature has a value very close to the threshold, a small change in thesignal may lead to a fingerprint bit with opposite value. The absolutevalue of the difference between feature value and threshold is used tomark each fingerprint bit as reliable or unreliable. The reliabilitiesare subsequently used to improve the actual matching procedure.

In this way, synchronisation may be obtained even though one of thesignals e.g. the video signal, has been obtained in a lesser quality,has been modified (e.g. compressed), etc.

Please note that the audio signal 101 and/or the video signal 103 may bea distorted version of the signal used during generation of thefingerprints, i.e. the signals of FIG. 1 a.

As mentioned in connection with FIG. 1 a, this embodiment may easily bemodified to accommodate synchronisation of more than two signals and/orsignals of another type than audio and/or video.

FIG. 2 illustrates a schematic block diagram of a fingerprint generationdevice according to the present invention.

Shown is fingerprint generation device 200 comprising a signal inputmodule 201, a fingerprinting module 202, a data base, memory storageand/or the like 203 communicating via a bus 205 or the like under thecontrol of one or more microprocessors (not shown). The fingerprintgeneration device 200 may in one embodiment optionally also comprise atransmitter and receiver 204 for communicating with other systems,devices, etc. via a wired and/or wireless network e.g. like theInternet.

The signal input module 201 receives a first 101 and at least a second103 signal. In the following two signals are received comprisingmultimedia content in the form of an analog or digital audio signal anda video signal. The input module 201 feeds the two signals to thefingerprinting module 202. The fingerprinting module 202 also receives arepresentation of the time points ( . . . , T_(n), T_(n+1), . . . ) thatare to be used as synchronisation time points. Alternatively, the timepoints are derived by the fingerprint generation device 200. If the timepoints are supplied and not generated by the fingerprint generationdevice 200 then it is not necessary to supply the fingerprint generationdevice 200 with the complete audio signal 101 and complete video signal103. It is then sufficient only to provide the respective segments ofthe audio signal 101 and video signal 103 that is used for thefingerprint generation, i.e. a segment of each signal for each timepoint.

The transmitter and receiver 204 may also be responsible for receivingone or more of the signals 101 and 103 and supply it/them to the signalreceiver 301 or directly to the fingerprint detector 302.

The fingerprinting module 202 computes a fingerprint on the basis of thereceived audio 101 and video 103 signals. A fingerprint may be derivedfor the entire content or for a part of the content. Alternatively,several fingerprints may be derived each from a different part.According to the present invention, a fingerprint is derived for eachtime point T_(n), T_(n+1), as explained in connection with FIG. 1 a.Alternatively, the fingerprinting module 202 may be divided into orcomprise two, e.g. distinct, fingerprint modules, one module forderiving audio fingerprints and one module for deriving videofingerprints.

The fingerprinting module 202 then supplies the computed fingerprintpair(s) to the database 203. As shown in FIG. 4, the database 203comprises video fingerprints organised by one column comprising videofingerprints 104 ‘V_FP 1’, ‘V_FP 2’, ‘V_FP 3’, ‘V_FP 4’, ‘V_FP 5’, etc.and corresponding audio fingerprints 102 ‘A_FP 1’, ‘A_FP 2’, ‘A_FP 3’,‘A_FP 4’, ‘A_FP 5’, etc.

The database 203 can be organised in various ways to optimise query timeand/or data organisation. The output of the fingerprinting module 202should be taken into account when designing the tables in the database203. In the embodiment shown in FIG. 4, the database 203 comprises asingle table with entries (records) comprising respective fingerprintpairs.

As mentioned, this exemplary embodiment may easily be modified toaccommodate synchronisation of more than two signals and/or signals ofanother type than audio and/or video.

FIG. 3 illustrates a schematic block diagram of a synchronisation devicedetecting and using fingerprints according to the present invention.

Shown is a synchronisation device 300 comprising a signal receiver 301,a fingerprint detector 302, a synchronisation circuit 303, a database,memory storage and/or the like 203 communicating via a bus 205 or thelike under the control of one or more microprocessors (not shown). Thesynchronisation device 300 may in one embodiment optionally alsocomprise a transmitter and receiver 204 for communicating with othersystems, devices, etc. via a wired and/or wireless network e.g. like theInternet.

The signal receiver 301 receives a first 101 and at least a second 103signal. In the following two signals are received comprising multimediacontent in the form of an analog or digital audio signal and an analogor digital video signal to be synchronised. The transmitter and receiver204 may also be responsible for receiving one or more of the signals 101and 103 and supply it/them to the signal receiver 301 or directly to thefingerprint detector 302.

The received signals are feed to the fingerprint detector 302 thatderives a fingerprint stream or segments thereof for each signal anddetermines if there are any matches with fingerprint pairs stored in thedatabase 203 as explained in connection with FIG. 1 b. If a match isfound then the specific synchronisation time point T_(n); T_(n+1) foreach signal are also determined. The specific determination of thesynchronisation time point T_(n), T_(n+1) for each signal is dependenton the scheme that have been used during generation of the audiofingerprint 102 and the video fingerprint 104 at that particular timepoint T_(n); T_(n+1).

Preferably, the specific synchronisation time point T_(n); T_(n+1) isgiven by letting the segment of the audio signal 101 and the segment ofthe video signal 103 that the matching fingerprint pair 102, 104originally has been based on during generation (according to FIG. 1 a)starting (substantially) at the given time point T_(n); T_(n+1). Inalternative embodiments, the segment of the audio signal 101 and thesegment of the video signal 103 end (substantially) at the given timepoint T_(n); T_(n+1), the segments of the audio and video signals 101,103 are starting or ending at a predetermined distance before or afterthe given synchronisation time point T_(n); T₊₁ or the givensynchronisation time point T_(n); T_(n+1) may be at a predetermined timepoint between a start and an end of the segments of the audio signal 101and the video signal 103.

The synchronisation device simply needs to be aware of the relationshipbetween a given fingerprint and the given time point used duringgeneration, which may be determined and implemented during manufactureof the synchronisation device or alternatively be updatable.

As explained, after a matching fingerprint pair 102, 104 is determined,the time point T_(n); T_(n+1) for each fingerprint of this pair is alsoknown (although not necessary its value but only its location in boththe audio and in the video signal) and serves as a synchronisation timepoint as these time points directly gives a reference point between thetwo signals 101 and 103. The synchronisation circuit 303 thencompensates for the delay or offset (if any) between the two signalse.g. by shifting one of them or both so that they are aligned withrespect to the synchronisation time point.

As a simple example, say that a synchronisation time point is at 5minutes and 34 seconds of the signals 101 and 103 during generationaccording to FIG. 1 a. During the generation of fingerprints (accordingto FIG. 1 a) one audio fingerprint would be derived at or near(depending on the used scheme) 5 minutes and 34 seconds in the audiosignal 101 and one video fingerprint would also be derived at or near(depending on the used scheme) 5 minutes and 34 seconds in the videosignal 103. These two fingerprints would then be stored and transmittedto a synchronisation device carrying out the synchronisation between thetwo signals. At the synchronisation device a fingerprint stream 105 ofthe audio signal and a fingerprint stream 106 of the video signal wouldbe compared against the stored two fingerprints. When a match betweenthe stored audio fingerprint and the audio fingerprint stream is found,then the location of the match (e.g. T n in 105 in FIG. 1 b) in thefingerprint stream gives the used synchronisation time point, i.e. whatshould correspond 5 minutes and 34 seconds. Likewise, when a matchbetween the stored video fingerprint and the video fingerprint stream isfound, then the location of the match (e.g. T n in 106 in FIG. 1 b) inthe fingerprint stream gives the used synchronisation time point, i.e. 5minutes and 34 seconds. The two signals may be shifted, but the exactlocation in the signals (as given by the segment of the fingerprintstream that matches a stored fingerprint) of what should be 5 minutesand 34 seconds may then be used to align the two signals. The specificvalue of the time point (5 minutes and 34 seconds) does not even need tobe known or derived specifically. The only knowledge needed is that thefingerprint matching location of the two signals 101; 103 should bealigned/synchronised. The synchronisation time point of 5 minutes and 34seconds may for example correspond to 5 minutes and 34 seconds in theaudio signal (e.g. because this is the original sound track used duringgeneration of the audio fingerprint) and to 6 minutes and 3 seconds inthe video signal (e.g. if the video signal further comprises commercialbreaks compared to the ‘original video signal used during the generationof the video fingerprint). The difference/offset between the twotime-values (6 min. 3 sec.−5 min. 34 sec.=29 sec.) may then be used tocompensate for the delay, e.g. by shifting the playback so that both theaudio signal and the video signal is played at the same time at thesynchronisation time point and forward (if no further modifications ofeither signals is present, e.g. an additional commercial break, etc.).

Preferably, the data layout of the database 203 corresponds to the oneshown in FIG. 4.

As mentioned, this exemplary embodiment may easily be modified toaccommodate synchronisation of more than two signals and/or signals ofanother type than audio and/or video.

FIG. 4 illustrates one example of tables/records according to thepresent invention. Shown is a table comprising fingerprint pairs 102,104. The table is, in this example, organised by one column comprisingvideo fingerprints ‘V_FP 1’, ‘V_FP 2’, ‘V_FP 3’, ‘V_FP 4’, ‘V_FP 5’,etc. and one column comprising the respective corresponding audiofingerprints 102 ‘A_FP 1’, ‘A_FP 2’, ‘A_FP 3’, ‘A_FP 4’, ‘A_FP 5’, etc.

FIG. 5 illustrates an alternative embodiment of a relationship betweentime points in a first and in a second signal. Shown are a first signal101 and a second signal 103. In this embodiment, a third or reference orcommon or internal time clock/line 107 (forth only denoted referencetime line) is also shown to better explain the principle of thisembodiment.

In this particular example of the alternative embodiment, a fingerprint(not shown) has been generated for the first signal 101 at a firstsynchronisation time point T_(n) having the value of 560. Thisparticular time point T_(n) for the first signal 101 is related to areference time frame as indicated by the reference time line 107 (asindicated by an arrow) to a time point having a value of 8:45:17.23(indicating that the first signal at T_(n)=560 should be presented at8:45:17.23) on the reference time line 107. A representation of thisindication or relationship between the particular time point T_(n) forthe first signal 101 (i.e. a first representation) may be associatedwith the generated first fingerprint and stored in a database (e.g. thesame or a different than the one containing the generated fingerprint)as will be explained later.

Further, a fingerprint (not shown) has been generated for the secondsignal 103 at a second synchronisation time point T_(m) having the valueof 1800. This particular time point T_(m) for the second signal 103 isalso related to the same reference time frame as indicated by thereference time line 107 (as indicated by an arrow) to a time pointhaving a value of 8:45:17.18 on the reference time line 107 (indicatingthat the second signal at T_(m)=1800 should be presented at 8:45:17.18).A representation of this indication or relationship between theparticular time point T_(m) for the second signal 101 (i.e. a secondrepresentation) may be associated with the generated second fingerprintand stored in a database (e.g. the same or a different than the onecontaining the generated fingerprint) as will be explained later.

The first and second representation may e.g. simply be the referencetime points of the first and second signal, respectively. In the aboveexample, the value 8:45:17.23 would then be stored with the fingerprintgenerated at T_(n)=560 and the value 8:45:17.18 would then be storedwith the fingerprint generated at T_(m)=1800.

During synchronisation, a synchronisation device according to thisembodiment generates a first and a second fingerprint stream orfingerprint segments as explained in connection with FIG. 1 b. Eachfingerprint stream (or segments) is compared with fingerprints, e.g.stored in a local or remote database, in order to determine if there isa match or not, also as explained in connection with FIG. 1 b. When amatching first and second fingerprint has been found then the first T n(i.e. 560 in the above example) and second synchronisation time pointsT_(m) (i.e. 1800 in the above example) are also known or derivable. Thenusing the above-mentioned first and second representation of therelationship to a reference time frame it is possible to determine howthe signals should be synchronised according to a given time frame.

As mentioned the first and second representation may be stored in one ormore databases and should be communicated to a synchronisation devicebefore synchronisation. In one embodiment the first and secondrepresentations are communicated directly to the synchronisation devicefor storage from a fingerprint generation device. Alternatively, thefirst and second representations are communicated to another device,e.g. a server, capable of communicating with a synchronisation device.This embodiment will be explained in greater detail in connection withFIG. 6.

FIG. 6 illustrates an embodiment where first and second representationsare stored at a remote location. Shown are an audio server 601 and avideo server 602 providing an audio stream and a video stream to anaudio fingerprint generator 202 and a video fingerprint generator 202,respectively. The audio and video fingerprint generator 202 functions asdescribed in connection with FIG. 2 and may be located in the samefingerprint generation device 200 or two different ones. In thisembodiment, the generated fingerprints are supplied to a database 203located at a (database) server 600 in communications connection with asynchronisation device 300. The server 600 also receives and stores afirst representation for each audio fingerprint and a secondrepresentation for each video fingerprint, as described e.g. inconnection with FIG. 5, i.e. the representations of the relationshipbetween time points of the audio and video streams and a commonreference time line or time frame.

The synchronisation device 300 functions as described in connection e.g.with FIGS. 3 and 5. It receives the audio and video stream to besynchronised from the audio and video servers 601, 602 and generates afingerprint stream or fingerprints segments of each and compares againstpredetermined fingerprints (corresponding to 102 and 104 of FIGS. 1 aand 1 b) signifying synchronisation time points, as described earlier.The predetermined fingerprints may be received from the fingerprintgeneration device 200 (as indicated by two broken arrows) or from theserver 600. If the predetermined fingerprints are received from theserver 600 storage is saved in the synchronisation device 300, which mayhave a more limited storage capability. The first and secondrepresentation for each fingerprint pair is preferably also receivedfrom the server 600 and is used in order to synchronise the audio andvideo stream before playback as described in connection with FIG. 5.

The server(s) may have stored predetermined fingerprints and/or theirassociated first and second representations for several different audioand video streams.

So, in one embodiment, the predetermined fingerprints are stored at thesynchronisation device 200 while the first and second representationsare stored at one or more servers 600. When a fingerprint pair has beendetected, the first and second representations of that pair aretransmitted from the server(s) and used in the synchronisation device200. Alternatively, all the first and second representation of allpredetermined fingerprints of a given audio and video streams may besupplied to the synchronisation device 200 before synchronisation isbegun, e.g. based on stream ID(s), etc.

In an alternative embodiment, the predetermined fingerprints along withtheir associated first and second representations are stored only at oneor more servers 600. Prior to the synchronisation of the streams boththe fingerprints and their associated first and second representationsare transmitted to the synchronisation device 200 e.g. based stream IDor the like. Alternatively, only the fingerprints are transmitted beforesynchronisation are begun and upon detection of matching fingerprintsthe associated first and second representations are transmitted to thesynchronisation device 200.

Please note that there will usually be a period time between thegeneration of fingerprints on the basis of the audio and video streamand when these streams are supplied to the synchronisation device 300.

The database 203 may be a single database or several databases that maybe located at a single or several servers.

FIG. 7 illustrates schematically more specifically how thesynchronisation in one embodiment may be done in a synchronisationdevice using buffers. Illustrated are a buffer 701 for buffering audiodata and a buffer 702 for buffering video data. For the audio buffer 701an in-pointer I-P indicates where the next audio sample, arriving fromthe audio stream, is to be placed in the buffer. An out-pointer O-Pindicates where the next audio sample is to be read. The out-pointermoves to the next slot at a pace set by a clock of the synchronisationdevice.

For the video buffer 702 an in-pointer I-P and an out-pointer O-P areshown that function in the same way as for explained for the audiobuffer 701.

Depending on a first representation (e.g. already present in thesynchronisation device or received from a server as explained earlier),the out-pointer is adjusted, i.e. shifted to earlier or later slot inthe buffer 701.

Likewise also for a second representation for the video buffer 702.

In this way, the out-pointers are adjusted on the basis of the first andsecond representations and thereby synchronise the out streams in a verysimple way.

In the claims, any reference signs placed between parentheses shall notbe constructed as limiting the claim. The word “comprising” does notexclude the presence of elements or steps other than those listed in aclaim. The word “a” or “an” preceding an element does not exclude thepresence of a plurality of such elements.

The invention can be implemented by means of hardware comprising severaldistinct elements, and by means of a suitably programmed computer. Inthe device claim enumerating several means, several of these means canbe embodied by one and the same item of hardware. The mere fact thatcertain measures are recited in mutually different dependent claims doesnot indicate that a combination of these measures cannot be used toadvantage.

1. A method, comprising: deriving a first fingerprint from a segment ofa first signal related to a first synchronization time point; deriving asecond fingerprint from a segment of a second signal related to a secondsynchronization time point; transmitting the first and secondfingerprints to a synchronization device, along with the first andsecond synchronization time points, or a relationship between the timepoints, wherein the first and the second signals are not derived fromeach other or a common source; and storing the first and secondfingerprints in association with the first synchronization time pointand the second synchronization time point, respectively, in at least onedatabase such that at least one of the first fingerprint or the secondfingerprint is searchable for comparison with a fingerprint streamderived from a third signal, the storing including refraining fromembedding the first fingerprint or the second fingerprint into the firstsignal or the second signal.
 2. The method of claim 1, wherein thetransmitting further comprises: transmitting the first and the secondfingerprints to the synchronization device via the Internet.
 3. Themethod of claim 1, wherein the segment of the first signal and/or thesegment of the second signal are unambiguously related to the firstand/or second synchronization time points according to a relationshipbetween the first and/or second synchronization time points and astarting time of the segments and/or an ending time of the segments. 4.The method of claim 1, wherein the first signal is an audio signal, andthe second signal is a video signal.
 5. The method of claim 1, whereinthe storing comprises: storing representations of timing relationshipsbetween the first and the second synchronization time points and areference time.
 6. The method of claim 5, further comprising:transmitting at least one of the fingerprints or the representations tothe synchronization device or a server communicatively coupled to thesynchronization device.
 7. A non-transitory computer readable mediumhaving stored thereon instructions for causing one or more processingunits to execute a method, comprising: generating a first fingerprintstream from a first signal; generating a second fingerprint stream froma second signal; comparing a segment of the first fingerprint streamwith one or more first fingerprints stored in at least one database inorder to determine if a match exists; comparing a segment of the secondfingerprint stream with one or more second fingerprints stored in the atleast database in order to determine if a match exists; and when a matchexists for both one of the first fingerprints and one of the secondfingerprints, determining a location of a first synchronization timepoint for the first signal and a location of a second synchronizationtime point for the second signal, and synchronizing the first and thesecond signal using the determined locations.
 8. The medium of claim 7,wherein the synchronizing comprises: delaying either the first or thesecond signal by an amount equal to a difference between the location ofthe first synchronization time point and the location of the secondsynchronization time point.
 9. The medium of claim 7, wherein thelocation of the first and/or the second synchronization time point aregiven by an unambiguous relation with a segment of the first signaland/or a segment of a second signal, respectively.
 10. The medium ofclaim 7, wherein the first and the second synchronization time pointsare the same.
 11. The medium of claim 7, wherein the first and thesecond synchronization time points are different, and wherein the methodfurther comprises: when the match exists for both one of the firstfingerprints and one of the second fingerprints, obtaining a firstrepresentation of a relationship between the first synchronization timepoint and a first time point of a reference time, obtaining a secondrepresentation of a relationship between the second synchronization timepoint and a second time point of said reference time, and using thefirst and the second time points of said reference time to synchronizethe first and the second signals, instead of determining the location ofthe first synchronization time point for the first signal and thelocation of the second synchronization time point for the second signal,and synchronizing the first and the second signal using the determinedlocations.
 12. The medium of claim 11, wherein the method furthercomprises: receiving the first and/or the second representation at asynchronization device from a server, and/or receiving the one or morefirst fingerprints and the one or more second fingerprints from theserver.
 13. The medium of claim 7, wherein the first signal is an audiosignal, and the second signal is a video signal.
 14. A system,comprising: a fingerprint generator to derive a first fingerprint from asegment of a first signal related to a first synchronization time point,and to derive a second fingerprint from a segment of a second signalrelated to a second synchronization time point, wherein the first andthe second signals are not derived from each other or a common source;and at least one database to store the first and second fingerprints inassociation with the first synchronization time point and the secondsynchronization time point, respectively, such that at least one of thefirst fingerprint or the second fingerprint is searchable for comparisonwith a fingerprint stream derived from a third signal, the storingincluding refraining from embedding the first fingerprint or the secondfingerprint into the first signal or the second signal.
 15. The systemof claim 14, further comprising: a synchronization device to synchronizeat least two signals comprising the first signal and the second signal.16. The system of claim 15, further comprising: a transmitter totransmit the fingerprints from at least one database to thesynchronization device.
 17. The system of claim 14, wherein the segmentof the first signal and/or the segment of the second signal areunambiguously related to the first and/or the second synchronizationtime points according to a relationship between the first and/or secondsynchronization time points and a starting time of the segments and/oran ending time of the segments.
 18. The system of claim 14, furthercomprising: a transmitter to transmit a representation of therelationship to a synchronization device, or to a server communicativelycoupled to the synchronization device; or a transmitter to transmit oneor more of the fingerprints to the server.